Chapter Nineteen Visual Generalization in Programming by Example

In programming by example (PBE; also sometimes called programming by demonstration) systems, the system records actions performed by a user in the interface and produces a generalized program that can be used later in analogous examples. A key issue is how to describe the actions and objects selected by the user, which determines what kind of generalizations will be possible. When the user selects a graphical object on the screen, most PBE systems describe the object using properties of the underlying application data. For example, if the user selects a link on a Web page, the PBE system might represent the selection based on the link’s HTML properties. In this chapter, we explore a different, and radical, approach—using visual properties of the interaction elements themselves, such as size, shape, color, and appearance of graphical objects—to describe user intentions. Only recently has the speed of image processing made feasible real-time analysis of screen images by a PBE system. We have not yet fully realized the goal of a complete PBE system using visual generalization, but we feel the approach is important enough to warrant presenting the idea. Visual information can supplement information available from other sources and opens up the possibility of new kinds of generalizations not possible from the application data alone. In addition, these generalizations can map more closely to the intentions of users, especially beginning users, who rely on the same visual information when making selections. Finally, visual generalization can sometimes remove one of the worst stumbling blocks preventing the use of PBE with commercial applications—that is, reliance on application program interfaces (APIs.). When necessary, PBE systems can work exclusively from the visual appearance of applications and do not need explicit cooperation from the API. 19.2 If You Can See It, You Should Be Able to Program It Every PBE system has what Halbert (1993) calls the “data description problem”: when users select an object on the screen, what do they mean by it? Depending on how you describe an object, it could result in very different effects the next time you run the procedure recorded and generalized by the system. During a demonstration to a PBE system, if you select an icon for a file foo.bar in a desktop file system, did you mean (1) just that specific file 372 y Command TNT Job Number: [002564] • Author: [Lieberman] • Page: 372 S